Computing systems and associated networks have greatly revolutionized our world ushering in what is now commonly called the “information age”. Efficient and effective storage of data is thus essential to reading, writing, persisting, and evaluating data. Often a file simply cannot fit onto a single storage device. In that case, the file is sharded (i.e., divided into parts) with each part (or shard) stored on a different storage device. This sharding process also helps to have parallel access to different shards of the file. Thus, fast data transfer enabled by parallel access is also a significant benefit of file sharding. Previous sharding technologies tend to address relatively uniform capacity constraints, such as throughput. For instance, a file might be sharded across similar hard drives. In high performance computing, a file might be sharded across specially designed networks in which throughput is designed to be uniform, regardless of the storage node.
As cloud computing services have developed, so too have storage services. A customer that establishes an account with a storage service may be guaranteed by a customer service agreement to have access to certain quantities of storage and/or certain I/O bandwidths. Of course, storage services hide from view the physical devices that are used to provide the storage on an account. Originally, such storage services were designed to simply store user data, such as documents, photographs, videos, music, and so forth.
The cloud is also able to access massive quantities of processing resources. Thus, complex data mining and analytics technologies have been developed. Such data mining or analytics often use as input massive quantities of data. Accordingly, storage services have become increasingly used as storage for massive amounts of raw data upon which analysis or data mining is performed.
The subject matter claimed herein is not limited to embodiments that solve any disadvantages or that operate only in environments such as those described above. Rather, this background is only provided to illustrate one exemplary technology area where some embodiments described herein may be practiced.